TRANSFORMER MODELS

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022.emnlp-main)

Challenge:	Recent studies have found that word alignments produced by the multi-head cross-attention weights are poor.
Approach:	They propose to introduce the hidden Markov model to the transformer architecture and introduce alignment components while keeping the system monolithic.
Outcome:	The proposed model outperforms the baseline model but is slower in training and decoding.

Copied to clipboard

Mor Geva, Avi Caciularu, Kevin Wang, Yoav Goldberg

HateCheck: Functional Tests for Hate Speech Detection Models (2021.acl-long)

Challenge:	Fig. 1 shows how feed-forward network (FFN) layers are utilized to build LMs.
Approach:	They reverse-engineer the operation of feed-forward network layers to find out how they work . they show that each update can be decomposed to sub-updates corresponding to single parameter vectors .
Outcome:	The proposed model reduces the toxicity of GPT2 by almost 50% and improves computation efficiency with a simple early exit rule, saving 20% of computation on average.

Copied to clipboard

Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, Janet Pierrehumbert

Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers (2025.tacl-1)

Challenge:	Hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score.
Approach:	They propose a suite of functional tests for hate speech detection models that measure model performance on held-out test data and then craft test cases to validate their quality.
Outcome:	The proposed tests show that the proposed models perform poorly on a small set of widely-used hate speech datasets.

Copied to clipboard

Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov

Fine-tuning Pre-trained Named Entity Recognition Models For Indian Languages (2024.naacl-srw)

Challenge:	Inductive biases in transformers can cause hierarchical generalization without explicitly encoding structural bias.
Approach:	They investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization.
Outcome:	The proposed model can generalize to novel syntactic forms without explicit bias . the proposed model is able to generalize on a dataset with a hierarchical grammar .

Copied to clipboard

Sankalp Bahad, Pruthwik Mishra, Parameswari Krishnamurthy, Dipti Sharma

Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks (2022.naacl-main)

Challenge:	Named Entity Recognition (NER) is a useful component in NLP applications.
Approach:	They propose to use annotated named entity corpora to classify a given entity into a category within a textual document.
Outcome:	The proposed model achieves an F1 score of 0.80 on an unseen dataset for Indian languages.

Copied to clipboard

Anton Chernyavskiy, Dmitry Ilvovsky, Pavel Kalinin, Preslav Nakov

Challenge:	Recent advances in machine learning have led to the use of contrastive loss for representation learning.
Approach:	They propose to use batch-softmax contrastive loss to train pairwise sentence embeddings . they propose to take a batch-softermax contrastitive loss and train it with different loss functions .
Outcome:	The proposed model improves on a number of datasets and pairwise sentence scoring tasks.

Multilingual Language Models Predict Human Reading Behavior (2021.naacl-main)

Copied to clipboard

Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, Lisa Beinborn

Layerwise universal adversarial attack on NLP models (2023.findings-acl)

Challenge:	Recent studies show that cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance.
Approach:	They compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing.
Outcome:	The proposed models predict reading time measures on Dutch, English, German, and Russian texts.

Copied to clipboard

Olga Tsymboi, Danil Malaev, Andrei Petrovskii, Ivan Oseledets

Shuffled-token Detection for Refining Pre-trained RoBERTa (2021.naacl-srw)

Challenge:	a new technique for layerwise UATs searches hidden layers of a network for universal adversarial triggers . a previous study showed that adversarials can fool models by perturbing samples that leave the ground truth label unchanged but can modify model prediction drastically.
Approach:	They propose a new approach to construct layerwise UATs by perturbing hidden layers of a network and propose LUATs that are more efficient than vanilla UAT methods.
Outcome:	The proposed method provides better transferability in a model-to-model setting with an average gain of 9.3% in fooling rate over baseline.

Copied to clipboard

Subhadarshi Panda, Anjali Agrawal, Jeewon Ha, Benjamin Bloch

Dataset Distillation with Attention Labels for Fine-tuning BERT (2023.acl-short)

Challenge:	State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks.
Approach:	They propose to refine a pre-trained NLP model by detecting shuffled tokens . they use a sequential approach to train a model using random shuffling .
Outcome:	The proposed model achieves better performance on 4 out of 7 GLUE tasks.

Copied to clipboard

Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, Manabu Okumura

AnnoPlot: Interactive Visualizations of Text Annotations (2024.eacl-demo)

Challenge:	Specifically, we propose to introduce attention labels, which can efficiently distill the knowledge from the original dataset and transfer it to the transformer models via attention probabilities.
Approach:	They propose to introduce attention labels which can efficiently distill the knowledge from the original dataset and transfer it to the transformer models via attention probabilities.
Outcome:	The proposed methods perform impressively in four different NLP tasks and achieve 93.2% accuracy in AGNews, which is 98.5% of the original dataset even with only one sample per class and only one gradient step.

Copied to clipboard

Elisabeth Fittschen, Tim Fischer, Daniel Brühl, Julia Spahr, Yuliia Lysa, Phuoc Thang Le

Challenge:	Annotation projects face challenges in data quality and validity, authors argue .
Approach:	They propose an open-source web application that analyzes, manages, and visualizes annotated text data.
Outcome:	The proposed application is open-source and promotes transparency and user control . it offers comprehensive views of span annotations and category systems without training or classification model .

N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses (2021.acl-short)

Copied to clipboard

Karthik Ganesan, Pakhi Bamdev, Jaivarsan B, Amresh Venugopal, Abhinav Tushar

Multimodality

Challenge:	Spoken Language Understanding systems parse spoken utterances into semantic structures like dialog acts and slots.
Approach:	They propose to use concatenated N-best ASR alternatives to represent utterances . they propose to employ a simpler utteration representation with no special delimiter .
Outcome:	The proposed model outperforms the prior state-of-the-art model on DSTC2 dataset.

Run LoRA Run: Faster and Lighter LoRA Implementations (2025.acl-industry)

Copied to clipboard

Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets

FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)

Challenge:	Existing studies on low-rank adapter training use the default chain of operations while calculating the output.
Approach:	They propose a framework that allows for efficient LoRA implementations by introducing low-rank adapters to linear layers and selecting the best forward and backward graphs based on FLOPs and time estimations.
Outcome:	The proposed framework significantly improves the speed of neural network training and fine-tuning with low-rank adapters.

Copied to clipboard

Rinat Khaziev, Usman Shahid, Tobias Röding, Rakesh Chada, Emir Kapanci, Pradeep Natarajan

Dialect Text Normalization to Normative Standard Finnish (D19-55)

Challenge:	Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error .
Approach:	They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions.
Outcome:	The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant.

Copied to clipboard

Niko Partanen, Mika Hämäläinen, Khalid Alnajjar

Syntactic Text Processing

Challenge:	a new method for normalizing dialect transcripts is proposed for normative Finnish . dialectal Finnish is the common way of communication for people online in finnish .
Approach:	They propose a method for normalizing dialectal Finnish into the normative standard Finnish.
Outcome:	The proposed method lowers the initial word error rate of the corpus from 52.89 to 5.73 . it can be used as one processing step with many types of spoken language materials.

TeluguNER: Leveraging Multi-Domain Named Entity Recognition with Deep Transformers (2022.acl-srw)

Copied to clipboard

Suma Reddy Duggenpudi, Subba Reddy Oota, Mounika Marreddy, Radhika Mamidi

LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models (2024.acl-demos)

Challenge:	Named Entity Recognition (NER) is a successful and well-researched problem in English due to the availability of resources.
Approach:	They propose to use two annotated NER datasets for the Telugu language . they compare the finetuned Telugus model with the existing model in NER .
Outcome:	The proposed models outperform existing models on a large dataset of 38,363 sentences on telugu and other languages.

Copied to clipboard

Abhishek Arora, Melissa Dell

PR-XAI: PageRank-Based Feature Attribution for Transformers (2026.acl-long)

Challenge:	Large language models (LLMs) are used for many computational analyses, but approximate string matching packages are not widely used in social science applications.
Approach:	The open-source package LinkTransformer provides an end-to-end software for performing record linkage and other data cleaning tasks with transformer LLMs.
Outcome:	The open-source package LinkTransformer outperforms standard methods in a variety of languages and settings.

Copied to clipboard

Behrooz Azarkhalili, Linyi Li, Maxwell W. Libbrecht

Improving Sentiment Analysis over non-English Tweets using Multilingual Transformers and Automatic Translation for Data-Augmentation (2020.coling-main)

Challenge:	Existing feature attribution methods for transformer models suffer from limitations that undermine their efficacy.
Approach:	They propose a feature attribution method for transformer models based on PageRank . they propose attribution methods that apply PageRank to attention-derived graphs .
Outcome:	The proposed method outperforms state-of-the-art methods in faithfulness and classification metrics with significant gains on long-form text.

Copied to clipboard

Valentin Barriere, Alexandra Balahur

Multilinguality

Challenge:	Existing models for sentiment analysis over tweets require a substantial amount of text to adapt to a domain where the syntax is different.
Approach:	They propose to use a multilingual transformer model to train over tweets in five different languages to adapt the model to non-English languages.
Outcome:	The proposed model improves over small corpora of tweets in non-English languages.

The Shape of Vulnerability: How Adversarial Perturbations Reshape the Topology of Language Model Latent Spaces (2026.acl-srw)

Copied to clipboard

Angelina Tsai, Shreya Subramanian, Catherine Liu, Kimberly Lopez, Leif Zinn-Brooks, Alexia E. Schulz, Adaku Uchendu

T2NER: Transformers based Transfer Learning Framework for Named Entity Recognition (2021.eacl-demos)

Challenge:	Large Language Models (LLMs) have unprecedented capabilities, but they pose security concerns . current adversarial attacks exploit vulnerabilities in the embedding space of language models, allowing attackers to bypass safety guardrails and cause significant harmful consequences.
Approach:	They propose to use topological data analysis to characterize how adversarial perturbations act on text inputs by computing persistent homology metrics from attention maps across different model architectures.
Outcome:	The proposed visualizations show that adversarial perturbations alter higher-dimensional topological features in ways that distinguish them from clean, non-adversarial inputs.

Copied to clipboard

Saadullah Amin, Guenter Neumann

Multimodal large language models for inclusive collaboration learning tasks (2022.naacl-srw)

Challenge:	Named entity recognition (NER) is an important task in information extraction due to large variations in entity names and flexibility in how entities are mentioned.
Approach:	They propose a Transformers based Transfer Learning framework for Named Entity Recognition (T2NER) that integrates transformer models with the state-of-the-art in NLP and provides a unified platform for transfer learning.
Outcome:	The proposed framework bridges the gap between the state-of-the-art in transformer models and the state of the art in NER with deep transformer models.

Copied to clipboard

Armanda Lewis

IrEne-viz: Visualizing Energy Consumption of Transformer Models (2021.emnlp-demo)

Challenge:	This project leverages advances in multimodal large language models to build an inclusive collaboration feedback loop for participants developing general collaboration skills.
Approach:	They propose to integrate advances in multimodal large language models into downstream tasks such as the learning analytics feedback loop.
Outcome:	The proposed model will be used to detect, model, and feedback participants developing general collaboration skills.

Copied to clipboard

Yash Kumar Lal, Reetu Singh, Harsh Trivedi, Qingqing Cao, Aruna Balasubramanian, Niranjan Balasubramanian

ferret: a Framework for Benchmarking Explainers on Transformers (2023.eacl-demo)

Challenge:	IrEne is an energy prediction system that accurately predicts inference energy consumption of transformer-based NLP models.
Approach:	They present an online platform for visualizing and exploring energy consumption of transformer-based NLP models.
Outcome:	The proposed system predicts energy consumption of transformer-based models and their components.

Copied to clipboard

Giuseppe Attanasio, Eliana Pastor, Chiara Di Bonaventura, Debora Nozza

Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems (2022.findings-acl)

Challenge:	Existing methods for interpreting transformer outputs are scattered and hard to operationalize.
Approach:	They propose a Python library to simplify the use and comparisons of XAI methods on transformers.
Outcome:	The proposed method provides better explanations and is preferable in the context of transformer models.

Copied to clipboard

Yi-Lin Tuan, Sajjad Beygi, Maryam Fazel-Zarandi, Qiaozi Gao, Alessandra Cervone, William Yang Wang

KIMERA: Injecting Domain Knowledge into Vacant Transformer Heads (2022.lrec-1)

Challenge:	Existing systems that require extensive labor to process user requests are limited in their reasoning capabilities and require extensive manual effort to design.
Approach:	They propose a method that allows a transformer model to walk on a large-scale knowledge graph to generate responses by reasoning over differentiable knowledge graphs.
Outcome:	The proposed method allows a transformer model to walk on a large-scale knowledge graph to generate responses.

Copied to clipboard

Benjamin Winter, Alexei Figueroa Rosero, Alexander Löser, Felix Alexander Gers, Amy Siu

LLM-FP4: 4-Bit Floating-Point Quantized Transformers (2023.emnlp-main)

Challenge:	Recent studies show that transformer models lack specific domain knowledge and are under-performing in broad domains like the medical domain.
Approach:	They propose a method for retraining and instilling attention heads with structured domain knowledge by masking redundant attention heads.
Outcome:	The proposed method improves on seven datasets in the medical domain in information retrieval and clinical outcome prediction settings.

Copied to clipboard

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng

VertAttack: Taking Advantage of Text Classifiers’ Horizontal Vision (2024.naacl-long)

Challenge:	Existing quantization solutions are integer-based and struggle with bit widths below 8 bits.
Approach:	They propose a method for quantizing weights and activations in large language models down to 4-bit floating-point values in a post-training manner.
Outcome:	The proposed method outperforms existing methods on common sense zero-shot reasoning tasks by 12.7 points.

Copied to clipboard

Jonathan Rusert

Text Classification

Challenge:	Current SOTA classifiers are subject to problems like bias and are vulnerable to adversarial attacks.
Approach:	They propose an attack to mimic a classifier's character based attack and thenrewrite those words vertically.
Outcome:	The proposed attack can drop the accuracy of 4 different transformer models on 5 datasets and preserve meaning.

Evaluating Transformer Models and Human Behaviors on Chinese Character Naming (2023.tacl-1)

Copied to clipboard

Xiaomeng Ma, Lingyu Gao

The Functional Relevance of Probed Information: A Case Study (2023.eacl-main)

Challenge:	Neural network models have been proposed to explain the grapheme-phoneme mapping process in humans for many alphabet languages.
Approach:	They propose to use a dictionary-like lookup procedure to map the letter strings to their pronunciations and then use 'transformers' to capture human behavior.
Outcome:	The proposed models learned the correspondence of the letter strings and their pronunciation, and captured human behavior in nonce word naming tasks.

Copied to clipboard

Michael Hanna, Roberto Zamparelli, David Mareček

Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)

Challenge:	Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences’ subjects and head verbs when performing subject-verb agreement.
Approach:	They propose to use probing to find out which words contain functionally relevant information encoded in the representations of subject plurality and words that agree with it in number in BERT.
Outcome:	The proposed model only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number.

Copied to clipboard

Fajri Koto, Jey Han Lau, Timothy Baldwin

TRACE: Training and Inference-Time Interpretability Analysis for Language Models (2025.emnlp-demos)

Challenge:	Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units.
Approach:	They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors.
Outcome:	The proposed model eliminates the decoder and reduces the search space for splitting points.

Copied to clipboard

Nura Aljaafari, Danilo Carvalho, Andre Freitas

Pragmatic Perspective on Assessing Implicit Meaning Interpretation in Sentiment Analysis Models (2025.acl-srw)

Challenge:	Existing tools for interpretability analysis of transformer models are post hoc, rely on scalar metrics or require nontrivial integration effort.
Approach:	They propose a modular toolkit for training and inference-time interpretability analysis of transformer models.
Outcome:	Experiments with autoregressive transformers show that TRACE reveals developmental phenomena overlooked by traditional scalar metrics such as loss or accuracy.

Copied to clipboard

Rashid Mustafin

Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences (2022.coling-1)

Challenge:	Using pragmatic theories of implicature, interpreting texts with implicit meaning correctly is essential for precise natural language understanding.
Approach:	They propose to use transformer models fine-tuned for sentiment analysis to illustrate the challenges in computational interpretation of implicatures.
Outcome:	The proposed model classifications reveal the limitations of supervised machine learning methods in detecting implicit sentiments.

Copied to clipboard

Jad Kabbara, Jackie Chi Kit Cheung

Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality (2025.acl-long)

Challenge:	Presuppositions are assumptions that are taken for granted by an utterance.
Approach:	They propose to use heuristics to create alternative "contrastive" test cases . they also analyze samples from ImpPres datasets to better understand their predictions .
Outcome:	The proposed model performs better on the ImpPres dataset than on the other datasets.

Copied to clipboard

Rahul Zalkikar, Kanchan Chandra

What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition (2021.tacl-1)

Challenge:	Innovative transformer-based language models produce contextually-aware token embeddings, but have been shown to encode unwanted biases for downstream applications.
Approach:	They extend previous work by evaluating social biases introduced after retraining an MLM under the masked language modeling objective and propose proxy functions within an iterative masking experiment to measure the quality of transformer models’ predictions.
Outcome:	The proposed proxy functions within an iterative masking experiment show that all transformer models encode concerning social biases.

Copied to clipboard

Piotr Żelasko, Raghavendra Pappagari, Najim Dehak

NLP Privacy Risk Identification in Social Media (NLP-PRISM): A Survey (2026.findings-eacl)

Challenge:	Existing punctuation in the transcripts has a massive effect on the models’ performance, and specific label set specificity does not affect dialog act segmentation performance.
Approach:	They apply two pre-trained transformer models to a conversation transcript as a sequence of dialog acts and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora.
Outcome:	The proposed models achieve 8.4% and 14.2% error rates on the Switchboard Dialog Act and Meeting Recorder Dialog Act corpora.

Copied to clipboard

Dhiman Goswami, Jai Kruthunz Naveen Kumar, Sanchari Das

How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection (2022.aacl-main)

Challenge:	Social media platforms such as X (formerly Twitter), Facebook, and Reddit generate user-generated content.
Approach:	They propose a framework to assess privacy risks in social media by evaluating vulnerabilities across six dimensions: data collection, preprocessing, visibility, fairness, computational risk, and regulatory compliance.
Outcome:	The proposed framework assesses privacy risks across six dimensions . it achieves F1-scores of 0.58–0.84, but incurs 1% - 23% drop under fine-tuning .

Copied to clipboard

Xiaomeng Ma, Lingyu Gao

The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing (2021.acl-long)

Challenge:	English past tense inflections is a typical quasi-regularity task, but it is criticized that it learns only to generalize the most frequent pattern, but not the regular pattern.
Approach:	They train a set of transformer models with different settings to examine their behavior on a typical English quasi-regularity task.
Outcome:	The models achieved high accuracy on unseen regular verbs and some accuracy on unseen irregular verbs.

Copied to clipboard

Ji Xin, Raphael Tang, Yaoliang Yu, Jimmy Lin

Mechanistic Analysis Of Universality: Numerical Comparison Circuits Across Transformer Architectures (2026.acl-srw)

Challenge:	Pre-trained language models have improved the state-of-the-art results on many NLP applications.
Approach:	They propose a simple error regularization trick that improves confidence estimation without substantially increasing the computation budget.
Outcome:	The proposed regularization improves confidence estimation without increasing computation budget.

Copied to clipboard

Arya Bhardia, Julian Ramirez, Siddhanta Verma, Karen Mkrtchyan

Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)

Challenge:	Mechanistic interpretability seeks to identify internal circuits within transformer language models but it is unclear whether they generalize across model families and scales.
Approach:	They propose to identify internal circuits within transformer language models by numerical comparisons.
Outcome:	The proposed model implementations are consistent across architecture and scale, the authors show . their results highlight the need for cross model comparisons to claim generalization of internal circuits.

Copied to clipboard

Tolulope Ogunremi, Dan Jurafsky, Christopher Manning

EM-PERSONA: EMotion-assisted Deep Neural Framework for PERSONAlity Subtyping from Suicide Notes (2022.coling-1)

Challenge:	AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks.
Approach:	They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data.
Outcome:	The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks.

Copied to clipboard

Soumitra Ghosh, Dhirendra Kumar Maurya, Asif Ekbal, Pushpak Bhattacharyya

Sentiment Analysis

Challenge:	Suicide continues to be one of the significant causes of death worldwide . EMotion-assisted personality subtyping is a novel approach to identify personality traits from suicide notes .
Approach:	They propose to use a PERSONAlity Detection Framework to identify personality traits from suicide notes and annotate them using a benchmark dataset.
Outcome:	The proposed method outperforms baselines on comprehensive evaluation using multiple state-of-the-art systems.

TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering (2025.acl-industry)

Copied to clipboard

Vinay Joshi, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

Discovering and Mitigating Indirect Bias in Attention-Based Model Explanations (2024.findings-naacl)

Challenge:	key-value caches in large language models consume memory, posing a major challenge for scalable deployment.
Approach:	They propose a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling.
Outcome:	The proposed technique reduces the KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy.

Copied to clipboard

Farsheed Haque, Depeng Xu, Shuhan Yuan

Applying Transfer Learning to German Metaphor Prediction (2024.lrec-main)

Challenge:	Discrimination is the unfair treatment or prejudice directed towards individuals, groups, or certain ideas or beliefs, intentionally or unintentionally.
Approach:	They propose an algorithm to detect and mitigate indirect bias in transformer models by leveraging attention explanations.
Outcome:	The proposed algorithm shows that it is more accurate than traditional fairness metrics and that it can be used to mitigate bias in transformer models.

Copied to clipboard

Maria Berger, Sebastian Michael Reimann, Nieke Marie Kiwitt

“Chère maison” or “maison chère”? Transformer-based prediction of adjective placement in French (2023.findings-eacl)

Challenge:	Existing methods for metaphor recognition are based on noun-verb pattern extraction or dictionary-informed.
Approach:	They propose to use an English corpus annotated for metaphor as a Gold standard for two different metaphor prediction setups.
Outcome:	The proposed method performs well on target-language data and achieves 90% F1 on target language data.

Copied to clipboard

Eleni Metheniti, Tim Van de Cruys, Wissam Kerkri, Juliette Thuilier, Nabil Hathout

Jigsaw Pieces of Meaning: Modeling Discourse Coherence with Informed Negative Sample Synthesis (2024.findings-eacl)

Challenge:	Prior work has shown that transformer-based language models are insensitive to permutated word order, but this is not the case with pretraining.
Approach:	They conduct experiments to assess whether transformer-based language models are able to learn the adjective position in noun phrases in French.
Outcome:	The proposed model is weaker with complex structures and fixed expressions, but favors context and global syntactic roles.

Copied to clipboard

Shubhankar Singh

Challenge:	Existing work on creating “informed” incoherent samples for coherence modeling has focused on permutations of a coherent document .
Approach:	They propose to use Constituency trees, Part-of-speech, semantic overlap to create “informed” negative samples that better represent or mimic incoherence.
Outcome:	The proposed methods improve the quality of the negative sample.

RED v2: Enhancing RED Dataset for Multi-Label Emotion Detection (2022.lrec-1)

Copied to clipboard

Alexandra Ciobotaru, Mihai Vlad Constantinescu, Liviu P. Dinu, Stefan Dumitrescu

IrEne: Interpretable Energy Prediction for Transformers (2021.acl-long)

Challenge:	RED is a machine learning-based resource developed for the automatic detection of emotions in Romanian texts.
Approach:	They propose an open-source extension of RED by adding trust and surprise . they propose two variants of ground truth suitable for multi-label classification and text regression .
Outcome:	The proposed model is based on two models with two transformer models, the Romanian BERT and the multilingual XLM-Roberta model, in categorical and regression settings.

Copied to clipboard

Qingqing Cao, Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian, Niranjan Balasubramanian

Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)

Challenge:	Existing software-based energy measurements of NLP models are not accurate because they do not consider the complex interactions between energy consumption and model execution.
Approach:	They propose an interpretable and extensible energy prediction system that predicts inference energy consumption of Transformer-based NLP models.
Outcome:	The proposed system predicts inference energy consumption of transformer models with an error of under 7% compared to the ground truth.

Copied to clipboard

Qinyuan Ye, Madian Khabsa, Mike Lewis, Sinong Wang, Xiang Ren, Aaron Jaech

Scheduled DropHead: A Regularization Method for Transformer Models (2020.findings-emnlp)

Challenge:	Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases.
Approach:	They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time.
Outcome:	The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings.

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, Ke Xu

Automated Refugee Case Analysis: A NLP Pipeline for Supporting Legal Practitioners (2023.findings-acl)

Challenge:	DropHead is a structured dropout method for regularizing multi-head attention . DropHed drops entire attention heads during training to prevent overfitting .
Approach:	They propose a structured dropout method specifically designed for regularizing multi-head attention mechanism . DropHead drops entire attention heads during training to prevent overfitting .
Outcome:	The proposed method can improve transformer models by 0.9 BLEU score on translation task and around 1.0 accuracy for various text classification tasks.

Copied to clipboard

Claire Barale, Michael Rovatsos, Nehal Bhuta

Information Retrieval

Challenge:	In Canada, retrieving similar cases and their analysis is a key part of legal work . long processing times are due to a significant backlog and to the amount of work required from counsels .
Approach:	They propose to extend existing neural named-entity recognition models to retrieve 19 categories of items from refugee cases.
Outcome:	The proposed pipeline achieves a superior F1- score on five of the targeted categories and superior to 80% on an additional 4 categories.

Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts (2022.findings-emnlp)

Copied to clipboard

Qinyuan Ye, Juan Zha, Xiang Ren

Event-Driven News Stream Clustering using Entity-Aware Contextual Embeddings (2021.eacl-main)

Challenge:	Pre-trained transformer models are capable of multitasking on diverse NLP tasks, but little is known about how multitaskability and cross-task generalization is achieved.
Approach:	They propose to use a transformer-based mixture-of-expert model with a router component to choose among experts dynamically and flexibly.
Outcome:	The proposed models improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks, and by 5.6% in zero-shot generalization settings.

Copied to clipboard

Kailash Karthik Saravanakumar, Miguel Ballesteros, Muthu Kumar Chandrasekaran, Kathleen McKeown

Text Clustering

Challenge:	a novel method for online news stream clustering is proposed . a user can scour the many news sources multiple times a day to find news articles .
Approach:	They propose a method for online news stream clustering that is a variant of the streaming K-means algorithm.
Outcome:	The proposed model achieves state-of-the-art on a standard stream clustering dataset of English documents.

Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages (2020.emnlp-main)

Copied to clipboard

Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, Dietrich Klakow

Modeling Intensification for Sign Language Generation: A Computational Approach (2022.findings-acl)

Challenge:	Recent studies show that results from high-resource languages cannot be easily transferred to realistic, low-resourced scenarios.
Approach:	They analyse performance of multilingual transformer models using available resources for Hausa, isiXhosa and NER and topic classification.
Outcome:	The proposed models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data.

Copied to clipboard

Mert Inan, Yang Zhong, Sabit Hassan, Lorna Quandt, Malihe Alikhani

Text Generation

Challenge:	End-to-end sign language generation models do not accurately represent prosody in sign language.
Approach:	They propose to model intensification in a data-driven manner to improve prosody in generated sign languages by modeling temporal and spatial variations.
Outcome:	The proposed models improve the prosody of generated sign languages by using data-driven models.

Do Syntax Trees Help Pre-trained Transformers Extract Information? (2021.eacl-main)

Copied to clipboard

Devendra Sachan, Yuhao Zhang, Peng Qi, William L. Hamilton

Syntactic Text Processing

Challenge:	Recent work suggests that incorporating syntax information from dependency trees can improve task-specific transformer models.
Approach:	They propose to incorporate dependency tree information into pre-trained transformers for three tasks . they propose a late fusion approach and a joint fusion technique to infuses syntax structure into attention layers.
Outcome:	The proposed models obtain state-of-the-art results on SRL and relation extraction tasks.

Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers (2025.findings-acl)

Copied to clipboard

Akhilesh Kakolu Ramarao, Kevin Tang, Dinah Baer-Henney

Syntactic Text Processing

Challenge:	A common generation task in morphology is morphological inflection, where a target form has to be generated from its corresponding lemma and feature tag.
Approach:	They propose to solve the Paradigm Cell Filling Problem (PCFP) by using encoder-decoder transformers to generate inflected verbs in Spanish.
Outcome:	The proposed model performs better on L-shaped verbs than regular verbs, but no consistent recency effects are observed.

Embedding Style Beyond Topics: Analyzing Dispersion Effects Across Different Language Models (2025.coling-main)

Copied to clipboard

Benjamin Icard, Evangelia Zve, Lila Sainero, Alice Breton, Jean-Gabriel Ganascia

Models Teaching Models: Improving Model Accuracy with Slingshot Learning (2023.eacl-main)

Challenge:	Using a literary corpus that alternates between topics and styles, we compare language models across French and English.
Approach:	They analyze how writing style affects embedding spaces across multiple language models . they use a literary corpus that alternates between topics and styles to compare their results .
Outcome:	The proposed model is based on two established literary works in French and English.

Copied to clipboard

Lachlan O’Neill, Nandini Anantharama, Satya Borgohain, Simon D. Angus

Challenge:	Existing methods to label datasets are expensive and require human labor . a semi-supervised method that augments a small dataset with labels reduces the cost of using simpler methods .
Approach:	They propose a semi-supervised method to augment a human-labeled dataset with labels from a teacher model to slingshot the performance of a student model.
Outcome:	The proposed method reduces the accuracy trade-off required to use simpler methods without disrupting their benefits.

Teacher and Student Models of Offensive Language in Social Media (2023.findings-acl)

Copied to clipboard

Tharindu Ranasinghe, Marcos Zampieri

Database reasoning over text (2021.acl-long)

Challenge:	Existing approaches to identify offensive language online use large pre-trained transformer models. however, the inference time, disk, and memory requirements of these models are prohibitively large.
Approach:	They propose to transfer knowledge from large transformer models to much smaller neural models to make predictions at the token- and post-level.
Outcome:	The proposed model performs 100 times better than transformer models but with 100 times less parameters and much less memory usage.

Copied to clipboard

James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, Alon Halevy

Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)

Challenge:	Existing models cannot handle database queries such as “List/Count all female athletes who were born in 20th century”.
Approach:	They propose a modular architecture to answer database-style queries over multiple spans from text and aggregate them at scale.
Outcome:	The proposed architecture scales to databases containing thousands of facts whereas current models are limited by how many facts can be encoded.

Copied to clipboard

Tharindu Madusanka, Riza Batista-navarro, Ian Pratt-hartmann

Character-level Language Models for Abbreviation and Long-form Detection (2024.lrec-main)

Challenge:	Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered.
Approach:	They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language.
Outcome:	The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research.

Copied to clipboard

Leonardo Zilio, Shenbin Qian, Diptesh Kanojia, Constantin Orasan

GradSim: Gradient-Based Language Grouping for Effective Multilingual Training (2023.emnlp-main)

Challenge:	Abbreviations and long forms are textual elements that are present in scientific communication . non-recognition of abbreviation and long form can lead to a negative impact on information retrieval .
Approach:	They propose to train and test language models for automatically identifying abbreviations and long forms . they use existing datasets annotated with abbrevations and their associated long forms to test them .
Outcome:	The proposed model can detect abbreviations and long forms on biomedical data . the proposed model improves on a previously untested dataset with biomedically-annotated datasets .

Copied to clipboard

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schuetze

MedNgage: A Dataset for Understanding Engagement in Patient-Nurse Conversations (2023.findings-acl)

Challenge:	Existing studies show that not all languages positively influence each other . multilingual training can help in those cases by sharing knowledge across languages .
Approach:	They propose a gradient similarity-based language grouping method for multilingual training that is better correlated with cross-lingual model performance.
Outcome:	The proposed method leads to the largest performance gains on a multilingual dataset and is better correlated with cross-lingual model performance.

Copied to clipboard

Yan Wang, Heidi Donovan, Sabit Hassan, Malihe Alikhani

Stanceformer: Target-Aware Transformer for Stance Detection (2024.findings-emnlp)

Challenge:	Literature suggests that actively engaged patients are more likely to obtain the full benefits of an intervention and exhibit better outcomes.
Approach:	They propose to annotate a dataset of patient-nurse conversations about cancer symptom management using a new framework for patient engagement.
Outcome:	The proposed model predicts patient-nurse conversations from socio-affective and cognitive dimensions.

Copied to clipboard

Krishna Garg, Cornelia Caragea

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding (2025.findings-naacl)

Challenge:	Existing transformer models that lack the capability to prioritize targets under-perform and are underperforming the task.
Approach:	They propose a target-aware transformer model that incorporates enhanced attention towards the targets during both training and inference.
Outcome:	The proposed model improves on state-of-the-art models and Large Language Models and can be used for other domains.

Copied to clipboard

Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, Alham Fikri Aji

Evaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition (2020.coling-main)

Challenge:	Multi-Layer Key-Value (MLKV) sharing reduces memory usage by 6x compared to Multi-Query Attention and Grouped-Query Attributes.
Approach:	They propose a novel approach that extends KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention and Grouped-Query Attributes.
Outcome:	The proposed approach reduces KV cache size by 6x with minimal performance loss and scales linearly with model size, batch size, and sequence length.

Copied to clipboard

Cedric Lothritz, Kevin Allix, Lisa Veiber, Tegawendé F. Bissyandé, Jacques Klein

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples (2022.coling-1)

Challenge:	Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP).
Approach:	They compare three transformer-based names to two non-transformer-based ones . they find transformer-derived models incrementally outperform non-tranformer models .
Outcome:	The proposed models outperform the studied models in most domains with respect to the F1 score.

Copied to clipboard

Venelin Kovatchev, Mariona Taulé

Can LLMs Find a Needle in a Haystack? A Look at Anomaly Detection Language Modeling (2025.findings-emnlp)

Challenge:	InferES is an original corpus for Natural Language Inference (NLI) in European Spanish .
Approach:	They propose to implement and analyze a corpus-creating strategy utilizing expert linguists and crowd workers to provide high-quality data and facilitate the systematic evaluation of automated systems.
Outcome:	The proposed model obtains 72.8% accuracy and performs moderately well on negation-based adversarial examples.

Copied to clipboard

Leslie Barrett, Vikram Sunil Bajaj, Robert John Kingan

Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning (2020.acl-main)

Challenge:	Anomaly detection (AD) is a problem in machine learning, but it is not always competitive on certain datasets.
Approach:	They propose a new approach to Anomaly detection based on large pre-trained language models in three modalities.
Outcome:	The proposed model beats baselines on anomaly detection when presented as imbalanced classification problem regardless of the concentration of anomalous samples.

Copied to clipboard

Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, Louise Naudin

Discovering Explanatory Sentences in Legal Case Decisions Using Pre-trained Language Models (2021.findings-emnlp)

Challenge:	Existing methods for solving common NLP tasks rely on fine-tuning of pre-trained transformer models.
Approach:	They propose a scoring method that casts a plausibility ranking task in full-text format without fine-tuning . they use masked language modeling head tuned during pre-training phase to exploit this method .
Outcome:	The proposed method produces strong baselines comparable to supervised approaches.

Copied to clipboard

Jaromir Savelka, Kevin Ashley

Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers (2025.findings-emnlp)

Challenge:	Understanding written laws is difficult because the abstract rules must account for a variety of situations, even those not yet encountered.
Approach:	They constructed a dataset of 26,959 sentences and labeled them in terms of their usefulness for explaining selected legal concepts.
Outcome:	The proposed models outperform the prior approaches and can learn surprisingly sophisticated features.

Copied to clipboard

Yingyu Liang, Heshan Liu, Zhenmei Shi, Zhao Song, Zhuoyan Xu, Jiale Zhao, Zhen Zhuang

Reasoning about Goals, Steps, and Temporal Ordering with WikiHow (2020.emnlp-main)

Challenge:	a large computational cost for attention computation in large language models is a major obstacle .
Approach:	They propose a convolution-like structure for attention computation using convolution matrices . they then propose an efficient approximation method to approximate the attention matrix .
Outcome:	The proposed method achieves nearly linear time complexity in n1+o(1) time.

Copied to clipboard

Li Zhang, Qing Lyu, Chris Callison-Burch

Reasoning

Challenge:	Existing datasets focus on relation between procedural events, but little attention has been paid to relation between events.
Approach:	They propose a set of reasoning tasks targeting goal-step relations and step-step temporal relations based on wikiHow articles . their automatically-generated training set allows models to transfer to out-of-domain tasks requiring knowledge of procedural events .
Outcome:	The proposed dataset improves on SWAG, Snips, and Story Cloze Test in zero- and few-shot settings.

Too Big to Fail: Larger Language Models are Disproportionately Resilient to Induction of Dementia-Related Linguistic Anomalies (2024.findings-acl)

Copied to clipboard

Changye Li, Zhecheng Sheng, Trevor Cohen, Serguei Pakhomov

Debiasing Multi-Entity Aspect-Based Sentiment Analysis with Norm-Based Data Augmentation (2024.lrec-main)

Challenge:	Existing studies show that the attention mechanism in transformer-based NLMs may present an analogue to the notions of cognitive and brain reserve.
Approach:	They propose a bidirectional ablation method that masks attention heads to display degradation of similar magnitude to masking in smaller models.
Outcome:	The proposed method exhibits properties attributed to the concepts of cognitive and brain reserve in human brain studies.

Copied to clipboard

Scott Friedman, Joan Zheng, Hillel Steinmetz

DeFaktS: A German Dataset for Fine-Grained Disinformation Detection through Social Media Framing (2024.lrec-main)

Challenge:	Recent research has explored strategies for reduce measurable biases in NLP predictions while maintaining prediction accuracy on held-out test sets.
Approach:	They propose to augment training data with norm-based language templates derived from previous language resources to reduce biases in NLP models.
Outcome:	The proposed model reduces topical bias to less than half while maintaining prediction quality on held-out test sets.

Copied to clipboard

Shaina Ashraf, Isabel Bezzaoui, Ionut Andone, Alexander Markowetz, Jonas Fegert, Lucie Flek

Ethical NLP

Challenge:	Distinctively curated across various news topics, DeFaktS offers an unparalleled insight into disinformation’s diverse characteristics.
Approach:	They propose to annotate every structural component and semantic element of a news piece, eliminating the need for external knowledge sources.
Outcome:	The proposed dataset contains 105,855 posts with 20,008 meticulously labeled tweets and eliminates the need for external knowledge sources.

ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation (2020.coling-main)

Copied to clipboard

Dario Stojanovski, Benno Krojer, Denis Peskov, Alexander Fraser

Multilinguality

Challenge:	Recent high scores on pronoun translation suggest current approaches work well . et al., 2018: are context-aware nmt models learning this task?
Approach:	They propose a test set to assess the ability to handle specific steps for pronoun translation . they propose heuristics that break down when translations require real reasoning .
Outcome:	The proposed model can model complex inferences required for translation of english into german . it shows that current approaches are not able to model all of this information well .

Crisis counselor language and perceived genuine concern in crisis conversations (2024.findings-emnlp)

Copied to clipboard

Greg Buda, Ignacio Tripodi, Margaret Meagher, Elizabeth Olson

Classical Sequence Match Is a Competitive Few-Shot One-Class Learner (2022.coling-1)

Challenge:	In the context of mental health interventions, an extensive body of research has found significant associations between therapists' behavioral traits and clinical effectiveness.
Approach:	They propose to extract linguistic features from crisis transcripts to analyze associations between therapist verbal behaviors and perceived genuine concern.
Outcome:	The proposed method could be used to automate real-time feedback to crisis counselors about clients' perceptions of the therapeutic relationship.

Copied to clipboard

Mengting Hu, Hang Gao, Yinhao Bai, Mingming Liu

Strong and Light Baseline Models for Fact-Checking Joint Inference (2021.findings-acl)

Challenge:	Existing models that use transformers are unable to learn new knowledge in the few-shot scenarios.
Approach:	They propose a few-shot one-class problem which takes a known sample as a reference to detect whether an unknown instance belongs to the same class.
Outcome:	The proposed method significantly outperforms transformer models under meta-learning and fine-tuning.

Copied to clipboard

Kateryna Tymoshenko, Alessandro Moschitti

Keyphrase Generation Beyond the Boundaries of Title and Abstract (2022.findings-emnlp)

Challenge:	Automated fact checking is rapidly gaining attention of the NLP and AI communities.
Approach:	They propose lightweight strong baselines for automated fact-checking systems . they propose to combine multiple pieces of evidence to verify a claim .
Outcome:	The proposed methods outperform heavier models on the leaderboard with blind TEST set.

Copied to clipboard

Krishna Garg, Jishnu Ray Chowdhury, Cornelia Caragea

Text Generation

Challenge:	Current approaches to keyphrase generation use only the title and abstract of the articles.
Approach:	They propose to integrate full text and semantically similar articles to generate keyphrases from a dataset that includes the full text of the articles along with the title and abstract.
Outcome:	The proposed model can generate keyphrases that are present or absent from the text.

Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell (2024.findings-emnlp)

Copied to clipboard

Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, Daniel Khashabi

Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks (2020.coling-main)

Challenge:	Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts.
Approach:	They propose to examine LLMs' long-context generalizations by probing their hidden representations.
Outcome:	The proposed models excel at processing extended contexts while preserving their positional bias.

Copied to clipboard

Trapit Bansal, Rishikesh Jha, Andrew McCallum

SHIELD: Defending Textual Neural Networks against Multiple Black-Box Adversarial Attacks with Stochastic Multi-Expert Patcher (2022.acl-long)

Challenge:	Pre-trained transformer models have shown great success in improving performance on downstream tasks, but fine-tuning on a new task still requires large amounts of labeled data.
Approach:	They propose a method which allows optimization-based meta-learning across tasks . they use transformers to train transformer models and find better generalizations .
Outcome:	The proposed method outperforms self-supervised training and pre-trained models on 17 NLP tasks.

Copied to clipboard

Thai Le, Noseong Park, Dongwon Lee

Gradient-based Adversarial Attacks against Text Transformers (2021.emnlp-main)

Challenge:	Existing methods to defend textual neural network models against adversarial attacks often require retraining and retrain . e.g., BERT, RoBERTa require great time and computation resources.
Approach:	They propose an algorithm that modifies and re-trains only the last layer of a textual NN and transforms it into a stochastic weighted ensemble of multi-expert prediction heads.
Outcome:	The proposed algorithm outperforms existing models against black-box attacks by 15%–70% . the proposed algorithm is based on a novel algorithm from software engineering .

Copied to clipboard

Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, Douwe Kiela

Domain-Agnostic Adapter Architecture for Deception Detection: Extensive Evaluations with the DIFrauD Benchmark (2024.lrec-main)

Challenge:	Existing methods for obtaining adversarial examples are difficult with text data.
Approach:	They propose a gradient-based adversarial attack against transformer models that searches for a distribution of adversarials parameterized by a continuous-valued matrix.
Outcome:	The proposed attack outperforms existing methods on a variety of natural language tasks with matching imperceptibility.

Copied to clipboard

Dainis A. Boumber, Fatima Zahra Qachfar, Rakesh Verma

Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering (2020.acl-main)

Challenge:	Existing research focuses predominantly on specific fields, which results in the need for clarity on linguistic markers associated with deception.
Approach:	They propose a domain-independent fraud detection benchmark with 100,000 honest and misleading statements in seven domains and a parameter-efficient finetuning adapter to improve tuning methods.
Outcome:	The proposed adapter outperforms all competition on the DIFrauD benchmark and is able to predict the performance of the proposed model.

Copied to clipboard

Changmao Li, Jinho D. Choi

Natural Language Interfaces

Challenge:	Existing approaches to embedding in multiparty dialogues are poor for span-based question answering (QA)
Approach:	They propose a novel approach to transformers that learns hierarchical representations in multiparty dialogue.
Outcome:	The proposed model improves on the FriendsQA dataset by 3.8% and 1.4% over the two state-of-the-art models.

CoMix: Guide Transformers to Code-Mix using POS structure and Phonetics (2023.findings-acl)

Copied to clipboard

Gaurav Arora, Srujana Merugu, Vivek Sembium

Multilinguality

Challenge:	Existing multilingual transformer models lack the ability to intermix words of one language into the structure of another.
Approach:	They propose a pretraining approach to improve representation of code-mixed data in transformer models by incorporating phonetic signals, a modified attention mechanism and weak supervision guided generation by parts-of-speech constraints.
Outcome:	The proposed model improves performance across four code-mixed tasks and generalizes on out-of-domain translation.

How to Leverage Digit Embeddings to Represent Numbers? (2025.coling-main)

Copied to clipboard

Jasivan Alex Sivakumar, Nafise Sadat Moosavi

Mitigating Confounding in Speech-Based Dementia Detection through Weight Masking (2025.acl-long)

Challenge:	Existing numerical reasoning models struggle to understand numbers, despite simple generalisations.
Approach:	They propose to use mathematical priors to compute digit embeddings and explicitly incorporate them into transformer models by adding a special token to the input embedded digits or introducing an additional loss function to enhance correct predictions.
Outcome:	The proposed method is compatible with any pretrained model and easy to implement.

Copied to clipboard

Zhecheng Sheng, Xiruo Ding, Brian Hur, Changye Li, Trevor Cohen, Serguei V. S. Pakhomov

Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitution (2022.emnlp-main)

Challenge:	Pre-trained neural language models fine-tuned on AD transcripts perform well, but little research has explored the effects of the gender of the speakers represented by these transcripts.
Approach:	They propose to use the Extended Confounding Filter and the Dual Filter to isolate and ablate weights associated with gender in dementia datasets.
Outcome:	The proposed methods overfit to training data distributions and disrupt gender-related weights, with the trade-off of slightly reduced dementia detection performance.

Copied to clipboard

Aiwei Liu, Honghai Yu, Xuming Hu, Shu’ang Li, Li Lin, Fukun Ma, Yawen Yang, Lijie Wen

Efficient Zero-shot Event Extraction with Context-Definition Alignment (2022.findings-emnlp)

Challenge:	Existing methods to attack transformer models are not effective at character level, but they are a natural attack scenario.
Approach:	They propose a character-level adversarial attack method against transformer models . they use a gradient-based method to find the most vulnerable words in a sentence .
Outcome:	The proposed method outperforms previous methods on sentence-level and token-level tasks.

Copied to clipboard

Hongming Zhang, Wenlin Yao, Dong Yu

One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them (2026.findings-acl)

Challenge:	Conventional supervised methods cannot generalize to event types out of the pre-defined ontology.
Approach:	They propose to use two separate transformer models to model the definition semantics of an event type name into the same embedding space and then minimize their embeddable distance via contrastive learning.
Outcome:	The proposed model outperforms all previous zero-shot EE methods with fast inference speed due to the disjoint design.

Copied to clipboard

Ali Kholmovaia, Paul Youssef, Nandi Schoots, Christin Seifert

Injecting structural hints: Using language models to study inductive biases in language learning (2023.findings-emnlp)

Challenge:	Knowledge editing methods such as ROME and MEMIT update factual associations by modifying MLP weights.
Approach:	They propose to use a mask to reverse edits by eliminating overattention in later layers . they also show that injecting the mask during editing drops editing success from 98% to 38% .
Outcome:	The proposed method reverses edits by eliminating overattention in later layers and drops editing success from 98% to 38%.

Copied to clipboard

Isabel Papadimitriou, Dan Jurafsky

Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension (2020.acl-main)

Challenge:	a recent study examines the cognitive inductive biases that make language learning possible.
Approach:	They structurally bias transformer language models by pretraining on synthetic data . they then evaluate their inductive biases by fine-tuning on three different languages .
Outcome:	The proposed method predisposes transformer models to three types of inductive biases . it also fine-tunes the models on three typologically-distant human languages .

Copied to clipboard

Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, Dong Yu

Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)

Challenge:	Existing approaches to machine reading comprehension (MRC) on long texts typically chunk text into equally-spaced segments without considering information from other segments.
Approach:	They propose to let a model learn to chunk in a more flexible way via reinforcement learning.
Outcome:	The proposed model extracts a text span from document and query as answer . previous models can only take a fixed-length (e.g., 512) text as input .

Copied to clipboard

Ramya Akula, Ivan Garibay

Meta-Learning Fast Weight Language Models (2022.emnlp-main)

Challenge:	Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity.
Approach:	They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods.
Outcome:	The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset.

Copied to clipboard

Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, Mohammad Norouzi

In Neural Machine Translation, What Does Transfer Learning Transfer? (2020.acl-main)

Challenge:	Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens.
Approach:	They propose a neural component that uses gradient updates as linear attention to improve model performance.
Outcome:	The proposed model can be applied at training time and learn to make good use of gradient updates.

Copied to clipboard

Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, Rico Sennrich

Evaluating Transformers for OCR Post-Correction in Early Modern Dutch Theatre (2025.coling-main)

Challenge:	a recent study found that word embeddings are not necessary for transfer learning.
Approach:	They perform several ablation studies that limit information transfer and measure the quality impact across three language pairs to gain a black-box understanding of transfer learning.
Outcome:	The proposed method can eliminate the need for a warm-up phase when training transformer models in high resource language pairs.

Copied to clipboard

Florian Debaene, Aaron Maladry, Els Lefever, Veronique Hoste

Multimodality

Challenge:	a new study examines the effectiveness of two types of transformer models for OCR post-correction in early modern Dutch plays.
Approach:	They propose to use large generative models and sequence-to-sequence models for OCR post-correction in early modern Dutch plays.
Outcome:	The proposed model outperforms generative models on the OCR post-correction task . the model outpersforms the model with the lowest error rate on the historical English dataset .

Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models (2024.emnlp-main)

Copied to clipboard

Michael Lan, Philip Torr, Fazl Barez

DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction (2024.emnlp-main)

Challenge:	Recent work aims to reverse engineer transformer models into human-readable representations . transformers exhibit strong capabilities on linguistic tasks, but their complex architectures make them difficult to interpret.
Approach:	They extend transformer models into human-readable representations that implement algorithmic functions by analyzing sequence continuation tasks.
Outcome:	The proposed model can be reverse-engineered into human-readable representations that implement algorithmic functions.

Copied to clipboard

Xueren Ge, Abhishek Satpathy, Ronald Williams, John Stankovic, Homa Alemzadeh

Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems (2021.emnlp-main)

Challenge:	Prior work focused on hierarchical label structures but neglected to incorporate external knowledge from medical guidelines.
Approach:	They propose to incorporate external knowledge from medical guidelines into domain knowledge enhanced classification for diagnosis prediction.
Outcome:	The proposed system outperforms state-of-the-art label-wise attention networks and transformer models on a real-world emergency medical services dataset and a public electronic health record dataset.

Copied to clipboard

Potsawee Manakul, Mark Gales

Tackling Irony Detection using Ensemble Classifiers (2022.lrec-1)

Challenge:	Training and inference using large transformer models can be computationally expensive because the self-attention's time and memory grow quadratically with sequence length.
Approach:	They propose a modified transformer architecture that constrains the encoder-decoder attention mechanism to a subset of input sentences while maintaining system performance.
Outcome:	The proposed architecture can be trained and inferenced using large transformer models with expensive training and induction costs.

Copied to clipboard

Christoph Turban, Udo Kruschwitz

Text Classification

Challenge:	Automated approaches to irony detection still fall short of what one would consider desirable performance.
Approach:	They propose to use transformer-based approaches to automate irony detection in social media . they propose to augmentation training data to address the binary and fine-grained problem .
Outcome:	The proposed methods improve performance over baselines and are not decisive for good results.

A Universal Avoidance Method for Diverse Multi-branch Generation (2026.findings-acl)

Copied to clipboard

Kyeongman Park, Minha Jhang, Kyomin Jung

Challenge:	generative models still lack human-level creativity, especially in multi-branch diversity tasks.
Approach:	They propose a model-agnostic and computationally efficient generation strategy that penalizes similarity among previously generated outputs.
Outcome:	The proposed method achieves 1.9 times higher diversity, runs 4.4 times faster, and requires only 1/64 of the FLOPs compared to state-of-the-art methods.

Do “English” Named Entity Recognizers Work Well on Global Englishes? (2023.findings-emnlp)

Copied to clipboard

Alexander Shan, John Bauer, Riley Carlson, Christopher Manning

Causal interventions expose implicit situation models for commonsense language understanding (2023.findings-acl)

Challenge:	Most of English named entity recognition datasets contain American or British English data . multiple problems may occur in low-resource English contexts, such as confusion of named entities with regionspecific meanings .
Approach:	They build a newswire dataset to analyze NER model performance on low-resource English variants . they find that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops .
Outcome:	The results show that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops.

Copied to clipboard

Takateru Yamakoshi, James McClelland, Adele Goldberg, Robert Hawkins

Reasoning

Challenge:	Classical psycholinguistic accounts have suggested that world knowledge enters into language understanding through structured schemas called situation models.
Approach:	They apply causal intervention techniques to transformer models to analyze performance on the Winograd Schema Challenge .
Outcome:	The proposed model performs well on the Winograd Schema Challenge .

Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection (2023.findings-emnlp)

Copied to clipboard

Pranaydeep Singh, Luna De Bruyne, Orphée De Clercq, Els Lefever

Leveraging Information Redundancy of Real-World Data through Distant Supervision (2024.lrec-main)

Challenge:	a negative emotion is a cognitive bias that affects how we express thoughts and opinions online . a recent study shows that negative words generate more engagement and clicks than positive ones .
Approach:	They propose to use readability and linguistic complexity metrics to better understand emotions . they propose to fine-tune three state-of-the-art transformers to detect emotions based on a dataset .
Outcome:	The proposed model fails to predict emotions on complex texts, the authors show . they also show that more advanced models fail to predict complex texts .

Copied to clipboard

Ariel Cohen, Alexandrine Lanson, Emmanuelle Kempf, Xavier Tannier

Challenge:	Existing methods for annotation of health care notes are promising but they are limited due to privacy regulations.
Approach:	They propose a text labeling method that leverages the redundancy of temporal information in a data lake to create a large programmatically annotated corpus and train transformer models using distant supervision.
Outcome:	The proposed method reduces expert annotation time, a scarce and expensive resource.

Approximating CKY with Transformers (2023.findings-emnlp)

Copied to clipboard

Ghazal Khalighinejad, Ollie Liu, Sam Wiseman

Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models (2026.findings-acl)

Challenge:	CKY algorithm is a cubic dependence on sentence length, but transformers can be used to approximate it.
Approach:	They propose a transformer-based approach that approximates the CKY algorithm by directly predicting a sentence's parse and avoiding its cubic dependence on sentence length.
Outcome:	The proposed approach achieves better performance than comparable parsers that make use of CKY, while being faster.

Copied to clipboard

Taisiia Tikhomirova, Dirk U. Wulff

Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning (2023.emnlp-main)

Challenge:	Existing studies show that transformer language models encode psychologically meaningful aspects of meaning at different depths.
Approach:	They conduct a layer-wise probing study of 58 psycholinguistic features across 10 transformer models . they find that apparent localization of meaning is method-dependent .
Outcome:	The results show that where meaning “lives” in transformer models reflects methodological choices and architectural constraints.

Copied to clipboard

Chong Li, Shaonan Wang, Yunhao Zhang, Jiajun Zhang, Chengqing Zong

Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)

Challenge:	Experimental results show that multi-head attention module evolves functional specialization after multi-task training.
Approach:	They propose a method to quantify the degree of functional specialization in multi-head attention . they propose 'multi-task training' method to increase functional specialisation and mitigate negative information transfer .
Outcome:	The proposed method increases functional specialization and mitigates negative information transfer in multi-task learning without adding any parameters.

Copied to clipboard

Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang

Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)

Challenge:	Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass.
Approach:	They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens.
Outcome:	The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost .

Copied to clipboard

Zhijin Guo, Chenhao Xue, Zhaozhen Xu, Hongbo Bo, Yuxuan Ye, Janet B. Pierrehumbert, Martha Lewis

Improving Online Job Advertisement Analysis via Compositional Entity Extraction (2025.emnlp-main)

Challenge:	Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction.
Approach:	a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities .
Outcome:	a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer.

Copied to clipboard

Kai Krüger, Johanna Binnewitt, Kathrin Ehmann, Stefan Winnige, Alan Akbik

SSA: Improving Performance With a Better Scoring Function (2026.acl-long)

Challenge:	Existing work on IE in OJAs has focused on skills extraction, but other information is extracted using job tasks, job titles, and work tools.
Approach:	They propose a compositional entity modeling framework for requirement extraction from online job advertisements (OJAs) they annotate a manually annotated dataset of 500 German job ads that captures roles, tools, experience levels, attitudes, and their functional context.
Outcome:	The proposed framework can extract requirements from a manually annotated dataset of 500 German job ads.

Copied to clipboard

Omar Naim, Swarnadeep Bhar, Jerome Bolte, Nicholas Asher

The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement (2024.lrec-main)

Challenge:	Despite the success of in-context learning, recent studies have identified systematic limitations in its generalization behavior.
Approach:	They propose a new attention scoring function that mitigates failures in transformer models . they use Scaled Signed Averaging to train the scoring function instead of Softmax .
Outcome:	The proposed scoring function outperforms transformer models with Softmax on NLP benchmarks and linguistic probing tasks.

Copied to clipboard

Jonathan Kamp, Lisa Beinborn, Antske Fokkens

Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction (2025.emnlp-main)

Challenge:	Existing methods for post-hoc explanations for transformer models disagree with each other . disagreement is often overlooked and the reasons for disagreement are not investigated .
Approach:	They propose to use a dynamic k approach to estimate syntactic spans to improve agreement between different methods.
Outcome:	The proposed method better agrees on syntactic span level, especially for the methods that agree the least with other methods.

Copied to clipboard

WonJin Yoon, Boyu Ren, Spencer Thomas, Chanhwi Kim, Guergana K Savova, Mei-Hua Hall, Timothy A. Miller

Summarization

Challenge:	Recent advances in large language models have enabled the automated processing of lengthy documents even without supervised training on a task-specific dataset.
Approach:	They propose a method for processing the summaries of long documents using different aspect-oriented prompts and integrate the information signals from these different prompts for supervised training of transformer models.
Outcome:	The proposed method improves on a high-impact task predicting readmissions from a psychiatric discharge using real-world data from four hospitals.

XAI-Attack: Utilizing Explainable AI to Find Incorrectly Learned Patterns for Black-Box Adversarial Example Creation (2024.lrec-main)

Copied to clipboard

Markus Bayer, Markus Neiczer, Maximilian Samsinger, Björn Buchhold, Christian Reuter

A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT (2026.findings-acl)

Challenge:	Adversarial examples can be used to trick machine learning models into making erroneous predictions, causing poorer insights and lower confidence in the information gathered.
Approach:	They propose a textual adversarial example method that identifies falsely learned word indicators by leveraging explainable AI methods as importance functions on incorrectly predicted instances.
Outcome:	The proposed method outperforms existing examples and training methods and shows baseline improvements of up to 23 percentage points on adversarial tasks.

Copied to clipboard

Louis Estève, Christophe Servan, Thomas Lavergne, Agata Savary